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EVALUATION  OF  THE  ADVANCED  SITUATIONAL  AWARENESS  TRAINING  (ASAT) 
PILOT  PROGRAM 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  Advanced  Situational  Awareness  Training  (ASAT)  program  was  evaluated  for 
adoption  into  the  U.S.  Army  Maneuver  Center  of  Excellence  (MCoE)  curricula  in  FY 14.  The 
ASAT  curriculum  is  designed  to  train  students  to  cognitively  evaluate  highly  complex 
environments,  focusing  largely  on  the  interpretation  of  nonverbal  behaviors  as  a  means  of 
improving  situational  awareness.  The  MCoE  Directorate  of  Training  and  Doctrine  (DoTD) 
requested  that  the  Army  Research  Institute  (ARI)  develop  instruments  by  which  Soldiers’ 
reactions  to  ASAT,  knowledge  of  ASAT  principles,  and  application  of  ASAT  principles  might 
be  measured  to  inform  the  decision  to  adopt  ASAT  into  the  curricula. 

Procedure: 

Three  separate  levels  of  course  evaluation,  loosely  based  on  Kirkpatrick’s  (2009) 
evaluation  model,  were  developed  and  administered  to  measure  the  effectiveness  of  ASAT 
training.  Measures  of  trainee  reactions,  two  separate  parallel  versions  of  academic-style  tests  of 
declarative  knowledge,  and  two  separate  parallel  video- vignette  tests  were  used  to  evaluate 
trainee  reactions,  knowledge,  and  skill  acquisition  resulting  from  exposure  to  ASAT  training. 
Each  measure  was  designed,  piloted,  refined,  and  then  implemented  to  both  address  performance 
standards  and  to  explore  changes  in  relevant  Soldier  skills  as  a  result  of  the  course. 

Findings: 

All  three  means  of  program  evaluation  yielded  support  for  the  effectiveness  and 
relevance  of  the  ASAT  program.  Soldier  reactions  exhibited  a  consensus  that  the  ASAT 
program  provided  some  of  the  most  relevant  and  valuable  training  yet  received  for  execution  of 
missions  in  support  of  Operation  Enduring  Freedom.  Soldier  declarative  knowledge  of  the 
ASAT  subject  matter  increased  significantly  as  a  result  of  exposure  to  the  ASAT  course.  ASAT- 
trained  Soldiers  appeared  to,  although  not  definitively  so,  exhibit  an  increased  ability  to  interpret 
nonverbal  behavior  in  order  to  identify  imminent  threats  when  viewing  real  footage  of  law- 
enforcement  scenarios. 

Utilization  and  Dissemination  of  Findings: 

The  findings  support  the  assertion  that  the  ASAT  program  improves  Soldiers’  ability  to 
detect  immediate  threats  and  in  that  manner  increases  Soldiers’  situational  awareness  in  the 
operational  environment.  The  findings  were  reported  to  MCoE  DoTD  to  help  inform  future 
decisions  regarding  the  continuation  of  this  program.  All  materials  were  made  available  to 
DoTD  for  the  purpose  of  continued  quality  control  efforts. 
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Evaluation  of  the  Advanced  Situational  Awareness  Training  (AS AT)  Pilot  Program 

Introduction 

The  asymmetric  warfare  environment  encountered  in  the  wars  in  Iraq  and  Afghanistan 
presented  U.S.  forces  with  an  enemy  that  wears  no  uniform,  has  no  official  national  affiliation, 
and  operates  from  no  centralized  base  of  operations.  In  order  for  our  forces  to  develop  an 
accurate  mental  representation  of  the  operational  environment,  they  must  be  able  to  identify  an 
enemy  that  hides  in  plain  sight  among  the  civilian  population.  The  Advanced  Situational 
Awareness  Training  (ASAT)  program  proposes  teaching  Soldiers  to  use  subtle,  nonverbal 
behavior  in  order  to  better  recognize  the  enemy  and  to  interpret  cues  that  can  reveal  his  or  her 
intended  course  of  action.  This  report  outlines  the  principles  taught  in  ASAT,  the  evaluation  of 
change  in  Soldier  performance  that  results  from  ASAT  training,  and  recommendations  regarding 
future  efforts  to  instruct  the  subject  matter  included  in  the  ASAT  program. 

Focusing  on  six  general  behavioral  domains,  ASAT  (see  Appendix  A)  diverges  markedly 
from  previous  situational  awareness  training  in  that  it  focuses  largely  on  the  interpretation  of 
nonverbal  behaviors  in  these  domains  as  a  means  of  improving  situational  awareness.  To  date 
three  versions  of  the  course,  a  4-hour  version,  a  5-day  version,  and  a  22-day  version,  have  been 
administered  to  Soldiers  at  Ft.  Fort  Benning  by  a  team  of  contractor  Subject  Matter  Experts 
(SMEs).  The  Et.  Benning  Maneuver  Center  of  Excellence  (MCoE)  Directorate  of  Training  and 
Doctrine  (DoTD)  tentatively  anticipates  that  the  ASAT  program  will  be  conducted  increasingly 
by  Army  cadre  during  2014.  This  report  details  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences’  (ARI)  research  in  developing  and  assessing  two  separate 
measures  of  performance  with  which  to  evaluate  the  relevant  skills  of  trainees  having  completed 
the  contractor-conducted  ASAT  course  and,  upon  its  initiation,  the  Army-conducted  ASAT 
course. 

Situational  Awareness 

The  Army  defines  situational  awareness  (SA)  in  EM  3-0  (Department  of  the  Army,  2008) 
as  “. .  .immediate  knowledge  of  the  conditions  of  the  operation,  constrained  geographically  and  in 
time.”  This  awareness  takes  the  form  of  mental  representation  of  an  individual’s  environment, 
informed  by  the  perception  and  comprehension  of  the  external  world  (Endsley,  2000).  In  a 
combat  environment,  a  Soldier’s  SA  is  necessarily  defined  in  terms  of  mission,  enemy,  terrain, 
troops,  time  available,  and  civilian  considerations  (METT-TC).  The  accuracy  of  this  mental 
representation  is  a  function  of  a  Soldier’s  ability  to  perceive,  comprehend,  and  make  projections 
informed  by  the  available  information.  Whereas  the  flow,  control  and  organization  of 
information  using  technological  systems  significantly  impacts  SA,  human  dimensions  have  equal 
impact  on  overall  SA  (Endsley,  1995). 

ASAT  is  designed  to  teach  Soldiers  skills  that  enhance  their  understanding  of  the 
operational  environment,  most  notably  the  identification  of  enemy  combatants  (insurgents)  and 
anticipation  of  likely  enemy  behavioral  courses  of  action.  To  illustrate  the  intended  contribution 
to  unit  readiness  of  the  ASAT  program,  it  is  necessary  to  consider  ASAT  techniques  in  the 
broader  context  of  SA.  Multiple  models  of  SA  have  been  developed  (Adams,  Tenney,  &  Pew, 
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1995;  Fracker,  1988;  Maggart  &  Hubal,  1999;  Salas,  Prince,  Baker,  &  Shrestha,  1995;  Smith  & 
Hancock,  1995;  Taylor  &  Selcon,  1994;  see  Endsley  et  ah,  2000  for  a  review);  however  the 
three-level  model  (Endsley,  1995)  is  the  most  commonly  cited  and  arguably  most  widely 
accepted  model  of  SA  (Salmon,  Stanton,  Walker,  &  Green,  2007).  Specifically,  the  Infantry- 
Eocused  SA  Model  (lESA;  Endsley  et  al.,  2000)  couples  the  current  conception  of  SA  with  the 
demands  specific  to  ground  combat  forces  and  includes  a  specific  focus  on  METT-TC.  The 
Endsley  lESA  will  be  used  herein  as  a  basis  for  discussing  the  role  of  ASAT  skills  in  developing 
SA  in  a  combat  environment. 

Endsley  (1995)  formally  defines  SA  as  “the  perception  of  the  elements  in  the 
environment  within  a  volume  of  time  and  space,  the  comprehension  of  their  meaning  and  the 
projection  of  their  status  in  the  near  future”  (p.  36).  The  model  focuses  on  the  product  of  SA  as 
opposed  to  the  process,  meaning  that  each  level  represents  a  complexity  and  comprehensiveness 
in  the  mental  representation  of  the  environment  rather  than  describing  the  mental  processes  at 
work  in  achieving  a  specific  level.  Endsley  outlines  three  levels  (see  below)  of  SA  along  with 
the  internal  and  external  influences  related  to  each  level.  These  levels  are  hierarchical  in  nature, 
meaning  that  lower  levels  are  required  in  order  to  form  higher  levels  of  SA  and  that  complexity 
and  sophistication  of  an  individual’s  mental  representation  of  the  environment  increase  moving 
to  higher  levels  of  SA.  At  each  stage,  the  mental  representation  of  the  situation  can  be 
influenced  by  either  direct  observation  of  events  or  organizational  communication  regarding 
events. 


Level  1  SA-  Perception.  Perception  is  the  most  basic  component  of  SA.  In  order  to 
construct  a  mental  representation  of  the  environment,  it  is  necessary  to  first  observe  and 
recognize  relevant  information.  Relevant  information  must  be  both  available  for  observation  and 
attended  to  in  order  to  facilitate  perception.  Previous  investigations  have  indicated  that  up  to 
76%  of  SA  errors  are  due  to  problems  at  the  perceptual  level  (Jones  &  Endsley,  1996). 

Individual  goals,  expectations,  training,  ability  and  experience  all  influence  the  perception  level 
of  SA.  In  order  to  obtain  Eevel  1  SA,  a  Soldier  must  be  aware  of  the  presence  and  nature  of 
crucial  information,  and  must  have  the  opportunity  to  observe  that  information.  At  this  level  of 
SA,  a  Soldier  might  notice  that  an  unfamiliar  local  is  standing  on  a  rise  several  dozen  yards  away 
from  an  upcoming  curve  in  the  road,  taking  note  that  he  is  wearing  overly  bulky  clothes  for  the 
time  of  year,  holding  a  small  electronic  device,  and  intently  watching  the  Soldier’s  convoy. 

Level  2  SA  -  Comprehension.  Eevel  2  SA  occurs  when  meaning  is  correctly  attributed  to 
information  perceived  in  the  environment.  Achieving  Eevel  2  requires  that  a  Soldier  correctly 
interpret  the  information  in  the  environment  after  becoming  aware  of  its  presence.  Information 
collected  must  be  correctly  identified  as  significant  or  non-significant,  and  then  used  to 
synthesize  a  higher-level  interpretation  of  the  scenario.  Twenty  percent  of  all  SA  errors  involve 
incorrect  comprehension  of  the  environment  after  the  accurate  perception  of  relevant  information 
(Jones  &  Endsley,  1996).  To  obtain  Eevel  2,  the  Soldier  should  be  able  to  correctly  conclude 
that  the  target  in  question  is  possibly  an  insurgent  and  that  an  attack  could  be  imminent. 

Level  3  SA-  Projection  of  Future  Status.  This  is  the  highest  level  of  SA  and  requires  that 
information  has  been  correctly  perceived  and  comprehended.  Eevel  3  requires  that  a  Soldier 
comprehends  not  only  the  immediate  situation,  but  also,  in  the  case  of  enemy  forces,  the  enemy’s 
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motivations,  limitations,  and  considerations  with  regards  to  that  Soldier’s  own  unit  in 
formulating  likely  courses  of  action.  This  level  of  understanding  generally  requires  a  high 
degree  of  experience  and/or  training  and  is  heavily  reliant  on  the  Soldier  possessing  an  extensive 
collection  of  schemas  against  which  to  compare  mental  representations  and  simulations  of  future 
events.  To  obtain  Level  3,  the  Soldier  should  be  able  to  recognize  that  the  curve  in  the  road  is  a 
likely  place  for  an  ambush  and  that  the  target  is  probably  either  a  trigger  man  or  a  spotter  in  a 
remote  lED  attack  that  could  be  initiated  when  that  Soldier’s  convoy  rounds  the  upcoming  curve. 
It  should  be  noted  that  to  achieve  Level  3  it  is  necessary  to  perceive  not  only  the  spatial  but  also 
the  temporal  relevance  of  projected  events;  our  Soldier  needs  to  able  to  not  only  project  the 
upcoming  ambush,  but  to  understand  that  it  is  imminent  and  contingent  upon  the  convoy’s 
movement  into  the  intended  kill  zone. 

Measuring  SA.  Numerous  instruments  have  been  developed  and  fielded  as  a  means  of 
measuring  SA  (see  Salmon  et  ah,  2007  for  a  recent  review).  Measurement  approaches  fall  into 
four  categories:  process  indices,  behavioral  measures,  performance  measures,  and  direct 
measures  (Endsley  et  ah,  2000).  Process  indices  evaluate  SA  during  the  assessment  process  and 
focus  on  measures  including  eye  tracking  and  intra-team  communication  that  reveal  to  what 
information  an  individual’s  attention  is  directed.  Behavioral  measures  focus  on  the  actions, 
application  of  force,  and  communication  of  information  that  result  from  SA  as  a  means  of 
estimating  the  accuracy  of  the  mental  representation  of  the  battle  space.  Performance  measures 
focus  on  the  doctrinal  appropriateness  of  tactical  maneuvers,  relative  lethality,  and  survivability 
of  a  unit  as  a  means  of  assessing  its  SA  in  understanding  the  battlefield.  Direct  measures 
evaluate  the  state  of  a  Soldier’s  understanding  of  the  combat  environment,  through  either  an 
objective  evaluation  of  knowledge  of  the  environment  or  a  subjective  self  or  observer  report  of 
knowledge. 

Direct  measures  of  SA  have  most  commonly  been  used  in  its  evaluation.  These  measures 
focus  on  SA  as  a  state,  while  largely  ignoring  the  processes  inherent  in  arriving  at  that  state,  and 
evaluate  the  accuracy  and  breadth  of  mental  representations  of  the  operational  environment. 
Focusing  on  direct  measures  has  allowed  researchers  to  capture  Soldiers’  representation  of  the 
environment,  the  most  critical  aspect  of  SA  and  not  specifically  addressed  by  any  of  the 
alternative  approaches.  Direct  measures  fall  into  two  broad  categories,  objective  and  subjective 
measures.  Objective  measures  evaluate  the  extent  to  which  mental  representations  match  the 
environment,  and  include  post-test  queries,  on-line  queries,  and  “freeze”  queries.  Post-test 
queries  introduce  minimal  interference  with  the  scenario,  but  suffer  from  memory  decay 
considerations.  On-line  queries  eliminate  memory  issues,  but  by  requiring  responses  during  a 
scenario  can  intrude  on  task  performance  and  alter  SA  by  shifting  attention.  “Freeze”  queries, 
including  the  Situational  Awareness  Global  Assessment  Technique  (SAG AT;  Endsley,  1988), 
stop  a  scenario  at  multiple  points  during  execution  in  order  to  determine  the  accuracy  of  mental 
representations  of  the  combat  environment  at  multiple  stages  within  the  same  context. 

Subjective  direct  measures  are  based  on  either  a  Soldier’s  or  an  observer’s  impression  of 
that  Soldier’s  or  unit’s  grasp  of  the  environment.  The  advantage  of  subjective  measures  is  the 
ease  with  which  they  can  be  used  in  Infantry  field  environment,  particularly  when  the  entirety  of 
the  state  of  the  environment  is  not  always  known.  The  main  disadvantage  is  that  these  measures 
use  no  objective  standard  for  evaluation.  Self-ratings,  such  as  the  Mission  Awareness  Rating 
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Scale  (MARS;  Matthews,  Beal,  &  Pleban,  2002),  the  Situational  Awareness  Rating  Technique 
(SART;  Taylor,  1990),  and  the  SA-Subjective  Workload  Dominance  scale  (SA-SWORD; 
Vidulich  and  Hughes,  1991)  ask  Soldiers  to  report  perceived  level  of  SA.  These  types  of 
measures  are  quickly  and  easily  administered,  offer  additional  data  that  complement  objective 
scenarios,  and  have  been  shown  to  correlate  with  performance  measures  in  aircraft  scenarios 
(Selcon  &  Taylor,  1990). 

Evaluation  Overview 

This  section  summarizes  the  preliminary  approaches  taken  to  evaluate  ASAT  and 
introduces  the  final  approach.  The  final  approach  is  detailed  in  the  “Methods”  section  below. 

The  methods  used  to  evaluate  the  effectiveness  of  the  ASAT  program  focused  on  the  skill 
set  taught  during  ASAT  and  not  on  its  influence  on  global  SA.  The  specific  focus  of  these 
evaluations  was  ASAT’s  effect  on  Soldiers’  interpretation  of  nonverbal  behavior.  Student 
evaluations  addressed  comprehension  of  course  material  and  student  ability  to  effectively  apply 
course  material  in  a  combat  profiling  exercise. 

Efforts  to  evaluate  the  effectiveness  of  the  ASAT  program  were  modeled  after  the  first 
three  levels  of  Kirkpatrick’s  four-level  framework  of  training  effectiveness  (Kirkpatrick,  1994). 
These  three  levels  include: 

1 .  Trainee  reactions  to  the  course, 

2.  Trainee  learning  during  the  course,  and 

3.  Transfer  of  training  to  job  relevant  tasks. 

Evaluation  efforts  emphasized  Eevels  2  and  3,  while  still  providing  basic  evaluation  of 
Eevel  1 .  Aspects  of  the  ASAT  evaluation  efforts  were  informed  by  the  previous  evaluation  of 
the  very  similar  Border  Hunter  Program  (Eautua  et  ah,  2010),  with  intent  to  capitalize  upon  that 
testing  effort.  The  evaluation  team  observing  the  Border  Hunter  Program  administered  eight 
separate  evaluation  instruments.  Included  in  the  program  were  a  cognitive  abilities  battery,  a 
declarative  knowledge  evaluation,  reactions  to  photographs,  situated  judgment  tasks  (SJTs), 
perceptual  aptitude  evaluations,  psychophysiological  measurements  (e.g.,  heart  rate),  behavioral 
observation  of  team  performance  during  the  field  problem,  and  reaction  surveys  regarding  the 
course.  Relative  to  the  Border  Hunter  evaluation,  one  of  the  objectives  of  the  ASAT  evaluation 
was  to  produce  a  more  economically  manageable  set  of  measurement  instruments. 

The  final  set  of  measures  used  to  evaluate  the  ASAT  program  effectiveness  included 
reaction  surveys,  tests  of  declarative  knowledge,  and  a  video  vignette  based  on  a  series  of  SJTs. 
One  objective  was  to  develop  evaluations  of  the  ASAT  course  that  maximized  the  diagnostic 
value  of  the  instruments  while  fitting  all  testing  activities  into  an  arbitrary  30-minute 
administration  window.  Considerations  shaping  the  development  of  the  three  measures  are 
described  and  explained  below. 

Level  1:  Reactions.  Trainee  reactions  to  the  course  were  evaluated  by  the  Et.  Benning 
MCoE  Office  of  Quality  Assurance  (QAO).  The  focus  of  these  evaluations  was  student  opinions 
regarding  the  quality  of  training  and  its  perceived  utility  in  increasing  mission  readiness.  These 


4 


types  of  evaluations  were  used  in  the  evaluation  of  both  the  USMC  Combat  Hunter  program 
(Kobus,  Palmer,  Kobus,  &  Ostertag,  2009)  and  the  Border  Hunter  program  (Fautua  et  al.,  2010) 
and  were  overwhelmingly  positive.  The  QAO-developed  evaluation  consisted  of  a  combination 
of  Likert-type  and  open-ended  items  that  asked  students  to  indicate  their  reactions  to  numerous 
aspects  of  the  ASAT  program.  Preliminary  responses  obtained  from  Army  ASAT  students  at  Ft. 
Benning  were  overwhelmingly  positive,  with  high  endorsement  for  the  quality,  relevance, 
applicability,  and  operational  value  of  the  ASAT  program. 

Level  2:  Learning.  In  order  to  evaluate  Soldiers’  learning  of  the  ASAT  training 
objectives,  ARI  focused  on  the  ability  of  ASAT  graduates  to  demonstrate  declarative  knowledge 
of  the  subject  matter.  Previously  administered  tests  of  declarative  knowledge  used  with  similar 
training  programs  were  utilized  to  inform  the  current  ASAT  evaluation  efforts.  Specifically,  the 
methods  of  evaluation  applied  to  the  Border  Hunter  Program  provided  a  starting  point  from 
which  to  develop  short-answer  evaluations  for  the  ASAT  program  (Fautua  et  ah,  2010).  Initial 
versions  of  each  test  were  piloted  and  refined  to  produce  the  final  versions. 

Level  3:  Transfer  to  job  relevant  tasks.  The  evaluation  of  trainee  ability  to  utilize  and 
perform  the  skills  taught  in  ASAT  in  order  to  interpret  the  operational  environment  underwent 
three  stages  of  development.  Tests  using  written  SJTs,  photograph-based  exams,  and  video  SJTs 
were  considered,  with  video  SJTs  being  the  final  selected  test  type.  Consideration  of  the  written 
SJTs  and  photograph-based  exams  and  their  disposition  are  summarized  below;  the  video  SJTs 
are  also  summarized  below  and  then  discussed  in  detail  in  the  following  sections. 

Situated  judgment  task  (SJT).  A  series  of  written  SJTs  heavily  informed  by  the  items 
administered  in  the  evaluation  of  both  the  Combat  Hunter  and  Border  Hunter  programs  were 
piloted  with  trainees  during  the  first  5-day  ASAT  course.  For  each  SJT  item,  a  written  paragraph 
provided  a  description  of  a  scenario  in  an  operational  environment.  Each  scenario  was  followed 
by  a  series  of  possible  courses  of  action.  Test  takers  responded  with  a  Likert-type  scale  to 
indicate  the  perceived  effectiveness  of  each  suggested  action.  After  the  first  pre-course 
administration  of  this  measure,  this  method  was  abandoned  with  SME  and  ARI  agreement  that 
the  task  offered  a  poor,  only  marginally  valid  representation  of  unprompted  real-time  decision 
making  in  the  operational  environment. 

Photograph  exam.  The  second  method  of  evaluating  situational  judgment  piloted  was 
that  of  presenting  trainees  with  a  series  of  four  slide  images  containing  subtle  but  significant 
pieces  of  information  relevant  to  interpretation  of  the  pictured  operational  environment. 

Trainees  were  asked  to  view  these  images  and  record  any  possible  interpretations  along  with 
supporting  evidence.  A  grading  rubric  was  developed  that  awarded  one  point  for  each  valid 
conclusion  and  piece  of  supporting  evidence  in  response  to  a  specific  photograph.  A  similar 
method  had  been  used  as  a  means  of  evaluating  the  Border  Hunter  program,  but  instead  focused 
on  the  use  of  descriptive  vs.  meaningful  language  in  response  to  a  series  of  photographs  (Eautua 
et  ah,  2010).  The  assessment  of  the  Combat  Hunter  course  also  used  a  picture  presentation 
methodology,  but  focused  on  the  quality  and  terminology  of  responses  (Kobus  et  al.,  2009). 

A  pre-training  and  a  post-training  photograph  exam  were  administered  to  a  pilot 
experimental  ASAT  class  and  to  a  comparable  control  group.  There  was  no  observed  increase  in 
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performance  among  AS  AT  trainees  as  compared  to  control  participants.  Also,  written  responses 
to  the  photographs  tended  to  be  terse  and  not  easily  categorized.  For  these  reasons  and  due  to 
agreement  among  both  SMEs  and  ARI  personnel  that  the  still  photograph  exam  was  not  a 
realistic  representation  of  the  dynamic  decision  making  process  in  the  operational  environment, 
the  use  of  the  picture  exam  was  abandoned. 

Video  SJT.  Two  video-based  situational  judgement  tests  designated  SJT-A  and  SJT-  B 
were  developed  to  provide  students  with  a  higher-validity  test  of  combat  profiling.  Video 
vignettes  have  been  used  in  a  variety  of  ways  to  include  medical  student  assessment  (Lievens, 
Buyse,  &  Sacked,  2005),  selection  of  hospitality  employees  (Jones  &  Decotiis,  1986),  the 
prediction  of  insurance  agent  turnover  (Delassio,  1994),  as  courtroom  evidence  in  cases  of 
suspect  confession  (Lassiter,  Ware,  Ratcliff,  &  Irvin,  2009)  and  for  assessing  Army  officer  social 
competence  (Schneider  and  Johnson,  2005).  Video-based  vignettes  are  also  widely  used  in 
screening  and  training  of  law  enforcement  officers  (Doerner  &  Nowell,  1999).  Evaluations  of 
video  vs.  written  vignette  formats  indicate  that  whereas  responses  to  written  vignettes  can  better 
predict  cognitive  performance,  responses  to  video-based  vignettes  are  better  able  to  predict 
interpersonally-oriented  criteria  (Leivens  &  Sacked,  2006).  Video  formats  are  less  influenced  by 
race  and  reading  comprehension  and  have  higher  perceived  validity  among  test  takers  when 
compared  to  paper  and  pencil  tests  (Chan  &  Schmitt,  1997). 

Significant  effort  was  initially  focused  on  obtaining  OEE  and  OIE  combat  footage  from  a 
variety  of  sources.  However,  most  footage  was  post-event;  no  suitable  footage  was  readily 
available  that  included  pre-event  indicators.  Therefore,  the  content  utilized  for  the  product  of 
both  video  exams  was  gathered  from  the  public  forum  Youtube.com  and  included  footage  of 
two-person  confrontations  in  public  settings,  convenience  stores,  and  police  stops,  all  from  North 
or  South  America.  The  content  was  all  related  to  daily  law  enforcement  activities  and  deemed  to 
be  the  closest  proxy  to  a  military  operational  environment  for  which  stimulus  material  was 
available.  Collection  focused  on  obtaining  video  clips  for  scenarios  that  culminated  in  violence 
and  scenarios  in  which  no  violence  occurred.  Details  of  evaluating  the  video  exams  follow 
below. 


Method 


Reaction  Survey  Development 

During  their  pre-deployment  training,  46  students  completed  reaction  surveys  following 
the  ASAT  class  conducted  for  4-73  CAV,  4*  BCT,  82"‘^  ABN  DIV  at  Et.  Bragg,  NC,  in  Eebruary 
2012.  The  reaction  survey  comprised  12  demographic  items  including  rank,  billet,  number  of 
deployments,  four  items  allowing  responses  on  a  Likert- type  scale,  four  open-ended  items 
specific  to  each  scale  item,  and  finally  an  additional  open-ended  remarks  section.  The  four 
scaled  items  asked  participants  to  rate  the  completed  training  relative  to  relevance,  importance, 
quality,  and  deployment  preparation  (Table  1). 
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Table  1 


Reaction  Survey  Items 

Item 

Rating  Scheme  for  All 
Items 

How  would  you  rank  the  relevance  of  this  training? 

5:  Outstanding 

How  would  you  rank  the  importance  of  this  training? 

4:  Excellent 

How  do  your  rank  the  quality  of  the  instructors? 

3:  Good 

In  comparison  to  other  pre-deployment  training,  how  important 
do  you  feel  ASAT  training  was  in  preparation  for  deployment? 

2:  Average 

1:  Below  Average 

0:  Poor 

Declarative  Knowledge  Test  Development 

The  ASAT  declarative  knowledge  tests  were  developed  based  on  materials  used  to 
evaluate  the  USMC  Combat  Hunter  program.  Tests  of  declarative  knowledge  administered  to 
Combat  Hunter  graduates  were  deconstructed  and  each  test  question  was  evaluated  for  relevance 
to  the  ASAT  training  objectives  and  for  item  integrity.  Personnel  from  ARI,  MCoE  Directorate 
of  Training  and  Doctrine,  and  the  contractor  team  all  inspected,  revised,  and  selected  items  until 
two  separate,  nominally  parallel  twenty-item  tests  emerged  for  piloting. 

The  tests  of  declarative  knowledge  were  designed  to  address  each  of  the  approved 
terminal  learning  objectives  (TLOs)  for  the  ASAT  course,  with  items  selected  to  ensure  that  each 
version  covered  comparable  TLOs.  The  declarative  knowledge  tests  covered  all  objectives  in 
ASAT  Unit  IB  (cognitive  processes)  with  the  exception  of  requiring  a  detailed  explanation  of  the 
steps  involved  in  decision  making.  Test  items  covered  objectives  from  Unit  1C  (perception) 
including  eye  function  and  aberrations,  change  blindness,  tunnel  vision  and  focus  lock,  and  the 
use  of  enhanced  observation  techniques.  Test  items  covered  objectives  from  Unit  2  (enemy 
decision-making)  including  urban  masking  and  the  seven-step  terrorist  planning  cycle.  Forty 
percent  of  the  items  on  each  test  focused  on  the  six  domains  of  combat  profiling.  Unit  3.  Both 
tests  evaluated  trainee  knowledge  of  biometrics,  kinesics,  proxemics,  geographies,  atmospherics, 
symbolism/iconography,  and  heuristics.  ASAT  Unit  1 A  is  an  introductory  session  explaining  the 
fundamental  concepts  of  ASAT  training  that  are  too  general  for  testing  using  multiple  choice  or 
fill-in-the-blank  methods  and  were  not  evaluated  by  the  tests.  Likewise,  Unit  4  focused  on 
implementation  of  all  learned  skills  during  the  field  problem  and  was  not  evaluated  by  the  tests 
of  declarative  knowledge. 
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Video  Exam  Development 


The  video  SJT  exams  were  developed  as  a  means  of  capturing  the  event  recognition  skills 
that  are  the  focus  of  training  for  the  ASAT  program.  These  exams  incorporate  content  loosely 
related  to  course  and  consistent  with  prior  SJT  examinations,  while  increasing  the  validity  of  the 
test  in  attempting  to  replicate  field  observations.  Two  types  of  test  items  were  produced:  one 
intended  to  measure  students’  ability  to  correctly  identify  scene  details  and  the  other  to  interpret 
scenarios  presented  in  each  video  clip. 

Content.  Several  hundred  clips  were  reviewed,  and  approximately  80  were  retained  for 
final  consideration  for  inclusion  in  the  exams.  Prior  to  final  selection,  all  clips  were  edited  to  a 
duration  of  at  least  10  seconds,  but  no  more  than  2  minutes.  Clips  that  culminated  in  violent 
events  were  edited  to  end  immediately  prior  to  the  event,  leaving  the  outcome  to  be  predicted  by 
the  observer.  Thematic  pairs  of  clips,  one  clip  with  a  violent  ending  and  the  other  with  a  non¬ 
violent  ending,  were  selected  and  non-systematically  assigned  to  one  of  two  exams  (Table  2). 
“Store  Footage”  clips  were  taken  from  various  security  camera  recordings,  “Police  Stop”  clips 
were  taken  from  various  law  enforcement  dashboard  camera  recordings,  and  “Confrontation” 
clips  were  various  recordings  of  two-person  interactions.  Exam  A  contained  one 
“Confrontation”  pair,  one  “Store  Footage”  pair,  and  two  “Police  Stop”  pairs.  Exam  B  contained 
one  “Confrontation,”  one  “Police  Stop,”  and  two  “Store  Footage”  pairs.  Each  clip  was  preceded 
by  a  brief  description  of  its  setting  to  help  orient  test  takers  to  the  scene. 

Table  2 

Video  Exam  Composition  by  Clip  Type 


Administration  Order 

Exam  A 

Exam  B 

Clip  1 

Store  Footage* 

Confrontation 

Clip  2 

Confrontation 

Store  Footage 

Clips 

Police  Stop 

Store  Footage* 

Clip  4 

Police  Stop* 

Police  Stop 

Clips 

Police  Stop 

Store  Footage* 

Clip  6 

Store  Footage 

Confrontation* 

Clip? 

Police  Stop* 

Police  Stop* 

Clip  8 

Confrontation* 

Store  Footage 

*Indicates  control  clip  (scenario  did  not  escalate  to  violence). 


Responses.  The  video  clips  were  projected  on  classroom  display  screens  mounted  at  the 
front  of  typical  MCoE  classrooms.  The  end  of  each  clip  was  immediately  followed  by  display, 
one  at  a  time,  of  three  multiple  choice  items  pertaining  to  that  clip.  Students  responded  using 
prepared  paper  answer  sheets  that  showed  only  question  numbers  and  answers  to  be  circled. 
Presenting  question  content  via  the  administration  video  prevented  students  from  previewing 
question  content  for  cues  about  what  in  the  clip  content  to  attend  to.  Each  multiple  choice 
question  contained  between  five  and  six  possible  statements  about  the  clip  and  included  the 
alternatives  “None  of  the  above”  and  “I  don’t  know.”  Students  were  instructed  to  select  all  true 
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statements  for  the  clip.  Thirty  seconds  were  allowed  for  each  question,  and  a  running  timer  was 
displayed  in  the  upper  left  hand  comer  of  the  administration  screen  that  counted  down  in  five 
second  increments.  When  the  timer  reached  zero,  the  next  question  was  displayed.  After  the 
timer  reached  zero  for  a  clip’s  third  question,  the  administration  video  proceeded  to  the  next  clip. 

Response  Content.  For  each  clip,  two  of  the  three  multiple  choice  items  addressed 
identification  of  details  (detail)  contained  within  that  clip  and  the  third  addressed  understanding 
of  the  scenario  (heuristic)  playing  out  during  a  clip.  For  each  test  item,  correct  responses  (hits) 
were  awarded  positive  points  and  incorrect  responses  (false  positives)  were  awarded  negative 
points.  The  number  of  points  (positive  or  negative)  for  any  response  were  assigned  relative  to 
how  important  that  response’s  information  was  in  the  interpretation  of  the  scenario.  For 
example,  a  heuristic  question  might  accme  six  positive  points  for  correctly  identifying  that  a 
stopped  motorist  was  a  serious  threat  and  exhibiting  immediate  signs  of  aggression,  but  only 
accme  one  point  for  correctly  identifying  that  the  motorist  stopped  because  of  a  road  barrier  vs.  a 
set  police  barrier.  Likewise,  negative  scores  indicated  how  serious  the  false  positive  responses 
were  in  misinterpretation  of  an  overall  scene,  with  misidentification  of  weapons  and  threats 
scored  with  the  highest  values  of  negative  points.  All  items  were  balanced,  so  that  the  sum  total 
of  all  an  item’s  responses  was  equal  to  0,  or  within  one  point  of  0.  Final  scores  were  calculated 
by  several  alternative  methods,  including  aggregation  of  all  positive  scores  only,  negative  scores 
only,  all  scores,  and  breakdowns  by  heuristic  and  detail. 


Results 


Reaction  Surveys 

Thirty-three  of  the  46  students  (72%)  self-identified  as  E4-E6.  Of  E4-E6  respondents,  27 
(82%)  reported  having  served  on  one  or  more  deployments.  The  second  most  represented  group 
was  officers,  with  seven  (15%)  identified  as  01-03,  three  of  which  (43%)  reported  having 
deployed.  Collectively,  all  other  rank  identifications  along  with  one  non-response  represented 
the  remaining  15%  of  students  (see  Table  3). 

Table  3. 

ASAT  Class  Composition  and  Deployment  History 


Rank 

No  Deployments 

1-1-  Deployments 

No  Response 

Total 

E1-E3 

3  (75%) 

1  (25%) 

4  (9%) 

E4-E6 

6  (18%) 

27  (82%) 

33  (72%) 

E7-E9 

1  (100%) 

1  (2  %) 

01-03 

4  (57%) 

3  (43%) 

7  (15%) 

No  response 

1  (100%) 

1  (2%) 

Total 

13  (28%) 

31  (67%) 

2  (4%) 

46  (100%) 

Note.  Percentages  in  either  the  total  column  or  the  total  row  represent  marginal  percentage  scores.  All  other 
percentages  are  computed  based  on  the  relative  percentage  of  deployment  history  response  for  each  rank. 
Blank  entries  indicate  that  no  responses  fell  into  a  particular  cell. 
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Among  all  students,  31  (67%)  reported  having  served  on  one  or  more  deployments,  13 
(28%)  reported  no  deployment  experience,  and  two  (4%)  failed  to  respond  to  this  item.  Among 
those  who  reported  deployment  experience,  18  (58%)  reported  having  served  on  one 
deployment,  eight  (26%)  as  having  served  on  two,  four  (13%)  as  having  served  on  three,  and  one 
(3%)  as  having  served  on  four  deployments. 

Overall,  responses  to  the  course  were  highly  positive.  Among  previously  deployed 
students,  88%  of  all  responses  to  the  scaled  items  were  a  five  of  five,  or  “Outstanding,”  (Table 
4).  Similarly,  among  students  with  no  deployment  history,  82%  of  responses  to  all  items  were  a 
five  of  five  (Table  5).  The  lowest  response  to  any  item  was,  “Good,”  indicated  by  one 
previously  deployed  student  on  item  4  and  indicated  once  for  item  2  and  once  for  item  4  among 
students  with  no  history  of  deployment. 

Table  4 

Previously  Deployed  Students’  Responses 


Item 

Rating 

Relevance 

Importance 

Quality 

Deployment 

Total 

5:  Outstanding 

28 

28 

28 

25 

109 

(90%) 

(90%) 

(90%) 

(81%) 

(88%) 

4:  Excellent 

3 

3 

3 

5 

14 

3:  Good 

2:  Average 

1 :  Below  Average 

0:  Poor 

(10%) 

(10%) 

(10%) 

(16%) 

1 

(3%) 

(11%) 

1 

(1%) 

Total 

31 

31 

31 

31 

124 

Note.  Two  participants  failed  to  respond  to  the  deployment  related  item  and  were  not  included  in  the 
analysis.  All  students  that  indicated  a  history  of  deployment  responded  to  every  question  item. 


A  series  of  analyses  was  conducted  to  determine  if  differences  existed  between  students 
with  a  history  of  deployment  and  those  with  no  history  of  deployment  in  their  responses 
to  the  scaled  items.  Due  to  the  observed  ceiling  effects  on  all  scaled  items,  it  was  inappropriate 
to  conduct  parametric  tests.  A  series  of  four  Chi  Square  analyses  were  conducted  to  test  for  a 
relationship  between  scale  response  and  deployment  history  on  every  scale  item.  There  was  no 
observed  relationship  between  the  responses  to  any  scaled  item  and  deployment  history.  The 
values  remained  non-significant  even  after  removing  the  Holm  correction  for  family-wise  error. 


10 


Open  responses  to  items.  Student  open  responses  both  to  the  four  reaction  survey  items 
and  to  the  instruction  in  general  were  uniformly  positive.  Representative  responses  are  given 
below: 

Item  1:  How  would  you  rank  the  relevance  of  this  training? 

“I  know  we  will  be  setting  in  OP’s  in  Afghanistan. .  .in  training  we  normally  do,  we 
just  sit  and  stare  at  nothing. .  .in  this  training,  we  actually  observed  things  around  us  to 
interact  with  our  environment.” 

“Your  tactics  can  be  used  to  get  to  a  reasonable  conclusion  of  what  is  going  on 
around  my  AO.” 

Table  5 

Responses  for  Students  with  No  Deployments 


Item 

Rating 

Relevance 

Importance 

Quality 

Deployment 

Total 

5:  Outstanding 

4:  Excellent 

11 

(85%) 

2 

II 

(85%) 

I 

13 

(100%) 

6 

(55%) 

4 

41 

(82%) 

7 

3:  Good 

(15%) 

(8%) 

I 

(36%) 

1 

(14%) 

2 

2:  Average 

(8%) 

(9%) 

(4%) 

1 :  Below  Average 

0:  Poor 

Total 

13 

13 

13 

11 

50 

Note.  Two  participants  failed  to  respond  to  the  deployment  related  item  and  were  not  included  in 
the  analysis.  Of  those  students  that  indicated  no  deployment  history,  two  failed  to  respond  to  item 
4. 


“With  the  mission  always  changing,  class  gives  you  different  ways  to  see  things.” 

“Anything  that  can  better  prepare  us. .  .especially  taking  into  consideration  our  need 
for  interactions  with  the  locals. . .” 
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Item  2:  How  would  you  rank  the  importance  of  this  training? 


“Not  knowing  the  language  has  smaller  borders  now.” 

“Would  be  best  used  as  a  whole  unit,  i.e.  platoons  or  whole  squads.” 

“It  put  all  [of]  the  pieces  together.  Things  you  do  all  the  time  but  never  think  of.” 

Item  3:  How  do  you  rank  the  quality  of  the  instructors? 

[Multiple  comments,  all  highly  positive;  the  primary  instructor  was  specifically 
named  multiple  times.] 

Item  4:  In  comparison  to  other  pre-deployment  training,  how  important  do  you  feel 
ASACD  training  was  in  preparation  for  deployment? 

“Nothing  will  compare  to  the  live  tissue  training  we  did  prior  to  this  deployment.” 

“I  feel  that  in  the  multi-dimensional  COIN  fight,  ASAT  is  just  as  important  as  the 
other  aspects  of  our  training,  kinetic  action  and  react  to  contact,  MEDEVAC  and  Eirst 
Responder,  and  EA  development.” 

[Multiple  comments  expressing  the  desire  that  this  training  had  been  available 
earlier,  before  previous  deployments.] 

Additional  Remarks 

“I  would  recommend  that  30-60  days  after  our  deployment  that  a  follow  up  is 
conducted  to  determine  if  this  training  was  truly  effective  and  sustainable  in  a  combat 
environment.” 

“. .  .1  would  like  a  copy  of  the  training  material  like  PowerPoint’s,  demonstrations, 
etc.  That  way  I  have  visual  reminders  of  this  skill  not  only  for  refreshers,  but  for 
future  training  of  my  soldiers. . .” 

“. .  .Eocus  more  towards  boots  on  the  ground  -  CPT  and  higher  don’t  need  it  as  much 
as  team  and  squad  and  platoon  leaders. . .” 

[Multiple  students  expressed  a  desire  to  attend  the  22-day  version  of  this  course.] 


Declarative  Knowledge  Tests 

Pilot  Administration  Results.  Pilot  administration  of  the  ASAT  declarative  knowledge 
tests  was  conducted  using  a  counterbalanced  design  between  the  two  separate  exam  versions. 
The  classes  included  in  this  pilot  analysis  began  on  21  Eebruary  2012  and  26  March  2012.  Pre- 
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course  administrations  occurred  prior  to  any  classroom  instruction  the  morning  of  the  first  day  of 
ASAT.  Post-course  administrations  were  conducted  after  all  classroom  training  at  the  end  of 
Day  3,  but  prior  to  any  field  training. 

The  21  February  ASAT  course  administered  the  exams  in  order  A  then  B  (Order  1),  and 
the  26  March  course  in  B  then  A  order  (Order  2).  A  total  of  37  students  enrolled  in  the  21 
February  class  completed  both  administrations,  and  a  total  of  57  students  in  the  26  March  class 
completed  all  materials.  Among  all  respondents  three  (3%)  were  female  and  the  remainder  were 
male.  The  majority  of  students  included  (over  90%)  were  1 1  series  MOS,  with  students  drawn 
from  the  Infantry  Advanced  Leader  Course  (ALC)  and  from  MCoE  training  support  units. 
Student  age  ranged  from  20  to  53  (mean  =  29.98,  SD  =  5.83),  and  years  of  service  ranged  from 
one  to  22  years  (mean  =  8.84,  SD  =  3.87).  Total  time  deployed  ranged  from  0  to  48  months 
(mean  =  27.67,  SD=  11.07). 

A  two  factor  Analysis  of  Variance  (ANOVA)  was  conducted  to  evaluate  the  effects  of 
presentation  order  and  test  version  on  student  performance.  The  results  support  that  the  two  tests 
are  comparable  measures  of  student  performance.  Planned  comparisons  revealed  that  there  was 
no  observed  difference  in  performance  between  Knowledge  Test  A  and  Knowledge  Test  B  when 
administered  prior  to  the  ASAT  course,  7^(1,  92)  =  3.52,  p  =  ns,  or  when  administered  after 
classroom  instruction,  7^(1,  92)  =  2.27,  p  =  ns.  A  main  effect  was  observed  for  test  version  that 
was  qualified  by  a  significant  interaction  between  test  version  and  administration  order,  7^(1,  92) 
=  212.68,  p  <  .05,  r|p^  =  .698.  Post  hoc  tests  revealed  that,  within  subjects,  scores  post  ASAT 
instruction  were  significantly  higher  than  were  scores  pre  ASAT  instruction  and  that,  between 
subjects,  for  both  test  versions,  scores  were  higher  when  the  knowledge  test  was  administered  at 
Time  2  (Table  6). 

Table  6 

ASAT  Test  Performance 


Raw  Scores 

Exam  A  at  Time  1 

Exam  B  at  Time  1 

Test  A 

7.73 

12.39 

Test  B 

13.05 

8.56 

Cicchetti  (1972)  interaction  post  hoc  tests  were  conducted.  Table  6  values  with  matching 
superscripts  exhibited  no  statistically  significant  difference.  Values  without  matching 
superscripts  exhibited  statistically  significant  differences  (Table  6).  All  scores  are  out  of  20 
possible  points. 
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Pre  ASAT  Post  ASAT 


■  Exam  A 

■  Exam  B 


Figure  1 .  Pilot  exam  raw  score  performance  before  and  after  ASAT. 


Reliability  was  calculated  for  each  scale  for  Time  1  and  Time  2  responses  from  all  ASAT 
students.  Both  Test  A,  Cronbach’s  alpha  =  .71,  and  Test  B,  Cronbach’s  alpha  =  .70,  exhibited  an 
acceptable  level  of  internal  stability.  A  frequency  analysis  of  both  exams  revealed  only  one  item 
to  be  problematic: 

Question:  Setting  up  an  observation  point  to  watch  a  targeted  individual  in  the  hopes  of 

gathering  additional  information  on  that  individual’ s  associates  is  an  example  of: 

Answer:  Tactical  Patience 

In  both  test  administrations,  only  one  trainee  correctly  answered  the  item.  This  item  was 
ultimately  retained  in  the  final  exam  versions  as  it  is  included  in  the  curriculum,  and  ARI  and 
SME  opinion  was  that  this  particular  point  had  been  passed  over  during  instruction  for  these  two 
test  administrations.  Minor  editorial  revisions  were  made  to  five  items,  and  the  exams  were 
otherwise  unchanged  from  pilot  to  final  versions. 

Final  Version  Administration  Results.  Einal  versions  of  the  ASAT  tests  of  declarative 
knowledge  Tests  A  and  B  were  administered  pre-  and  post-course  to  304  ASAT  program 
participants.  Versions  were  administered  in  a  counterbalanced  manner  in  order  to 
simultaneously  test  for  learning  and  equivalence  of  tests.  All  participants  were  male,  and  the 
majority  were  E5/E6  (52%)  or  01  (45%).  Eor  all  analyses,  rank  was  coded  as  Enlisted  (55%)  vs. 
Officer  (45%).  Table  7  provides  an  Enlisted  vs.  Officer  comparison  of  all  relevant  demographic 
variables. 
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Table  7 


Officer  and  Enlisted  Demographics 


Officer 

Enlisted 

Variable 

Mean  (SD) 

Min 

Max 

Mean  (SD) 

Min 

Max 

Age 

24.5 

(3.47) 

21| 

39 

29.3 

(4.51) 

21 

46 

Years  in  Service 

2.7 

(3.27) 

o| 

19 

8.3 

(3.70) 

2 

25 

Months  Deployed 

2.7 

(7.44) 

o| 

42 

26.2 

(11.4) 

0 

56 

Twenty-one  Enlisted  and  22  Officers  reported  having  experienced  some  level  of  exposure 
to  ASAT-related  material  prior  to  the  course  and  were  removed  from  further  analyses,  leaving 
116  Officers  (44%)  and  145  Enlisted  (56%)  in  the  final  sample  (Table  8).  The  final  sample 
comprised  almost  exclusively  IN  AEG  (MOS  1 IB/C,  52%)  and  Armor  Basic  Officer  Eeader 
Course  (ABOEC)  students  (CE  19A,  43%);  thus  MOS  and  CE  were  not  included  in  further 
analyses. 


Table  8 

Final  Sample  Distribution  * 


Administration  Order 


A  then  B 

B  then  A 

Total 

Enlisted 

105 

40 

145 

Officer 

32 

84 

116 

Total 

137 

124 

261 

*  Second  exam  versions 

Note.  Forty-three  participants  were  removed  who  reported  some  prior  level  of  AS  AT  exposure. 


Three  variables-age,  time  in  service,  and  months  deployed-were  considered  for  inclusion 
as  a  covariate  in  further  analyses.  As  all  three  variables  were  significantly  related  (bivariate 
Pearson  r  values  ranging  from  .54  to  .81),  a  single  covariate  was  chosen  for  entry  into  the  model. 
Time  in  service  was  negatively  related  to  raw  scores  from  both  Exam  A,  r  =  -.32,  p  <  .01,  and 
Exam  B,  r  =  -.12,  p  =  .06,  but  suffered  from  non-normality  and  multiple  extreme  outliers.  A 
natural  log  transformation  was  conducted  on  time  in  service  values  after  adding  one  to  each 
value  as  a  means  of  avoiding  computational  problems  associated  with  values  equal  to  zero.  Post 
transformation  values  exhibited  no  outliers  and  acceptable  skew  and  kurtosis  values. 
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Both  exams  were  analyzed  for  internal  consistency  using  the  final  sample.  Both  Exam  A 
(a  =  .76)  and  Exam  B  (a  =  .61)  exhibited  acceptable  levels  of  internal  consistency.  Neither  exam 
contained  an  item  answered  either  correctly  or  incorrectly  by  all  participants. 

An  analysis  of  covariance  (ANCOVA)  was  conducted  to  assess  learning  that  occurred 
between  pre-  and  post-course  test  administrations  after  adjusting  for  time  in  service.  Rank 
(Officer  vs.  Enlisted)  and  test  administration  order  were  entered  as  between  subjects  variables, 
test  as  a  repeated  measure,  and  log  transformed  time  in  service  as  a  covariate.  Mean  scores  are 
shown  in  Table  9. 

Table  9 

Estimated  Marginal  Means 


Enlisted 

Officer 

10.29  (.25) 

11.75  (.30) 

Exam  A  then  B 

Exam  B  then  A 

Exam  A  then  B 

Exam  B  then  A 

Exam  A 

Exam  B 

Exam  B 

Exam  A 

Exam  A 

Exam  B 

Exam  B 

Exam  A 

Mean 

7.13 

13.68 

9.73 

10.95 

8.26 

14.05 

11.44 

13.26 

(SFl) 

.30 

.34 

.47 

.42 

.50 

.56 

.38 

.34 

Six  planned  comparisons  were  conducted  in  order  to  test  specific  hypotheses  regarding 
the  nature  of  anticipated  interactions.  After  collapsing  across  counterbalance  administration 
order  and  rank,  post-test  scores  (adjusted  mean  =  12.95)  were  significantly  higher  than  pre-test 
scores  (adjusted  mean  =  9.06),  F(l,  256)  =  413.54,  p  <  .05.  As  shown  in  Eigure  2,  when 
administered  as  a  pre-test,  student  scores  on  exam  B  (adjusted  mean  =  10.583)  were  significantly 
higher  than  were  scores  on  Exam  A  (adjusted  mean  =  7.696).  Similarly,  post-test  scores  were 
higher  when  Exam  B  was  administered  as  a  post-test  (adjusted  mean  =  13.707)  than  when  Exam 
A  was  administered  (adjusted  mean  =  12.105),  E(l,  256)  =  124.31,  p  <  .05.  Officers 
outperformed  enlisted  (Eigure  3)  on  pre-tests  (enlisted  adjusted  mean  =  7.85;  officer  adjusted 
mean  =  10.56),  F(l,  256)  =  23.26,  p  <  .05,  and  on  post-tests  (enlisted  adjusted  mean  =  12.70; 
officer  adjusted  mean  =  13.48),  E(l,  256)  =  38.27,  p  <  .01.  There  were  no  observed  differences 
between  enlisted  and  officers  in  learning  from  pre-course  to  post-course  administration,  E(l, 

256)  =  .032,p>.10. 

The  covariate,  log  transformed  time  in  service,  was  not  significant  among  between 
subjects  factors  or  within  subjects  factors.  A  main  effect  was  observed  for  rank,  with  officers 
(adjusted  mean  =  1 1.75)  scoring  higher  than  enlisted  (adjusted  mean  =  10.29),  E(l,  256)  =  9.99, 
p  <  .01.  A  main  effect  was  also  present  for  test,  with  higher  test  scores  observed  for  Exam  B 
(adjusted  mean  =  1 1.34)  than  for  Exam  A  (adjusted  mean  =  10.70)  when  collapsed  across  all 
other  factors,  E(l,  256)  =  18.52,  p  <  .01.  There  was  an  observed  main  effect  for  counterbalance 
order,  E(l,  256)  =  4.65,  p  <  .05,  qualified  by  an  exam  by  counterbalance  interaction,  F(l,  256)  = 
326.87,  p  <  .01,  in  which  exam  scores  depended  upon  administration  order  (Eigures  2  and  3). 
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Order  1 
•  Order  2 


Figure  2.  Performance  on  Exams  A  and  B  by  administration  order.  Order  1 
is  A  then  B;  Order  2  is  B  then  A. 


—  —  Enlisted 
Officer 


Figure  3.  Pre-  and  post- test  scores  for  Officer  and  Enlisted  collapsed  across  exam 
versions  A  and  B. 


Video  Vignette  Situational  Judgment  Tests 

A  total  of  175  Soldiers  completed  both  pre-course  and  post-course  administrations  of  the 
video  vignette  examination.  The  examinations  were  administered  to  two  Infantry  AEG  classes 
and  to  two  ABOEC  classes,  counterbalanced  for  order.  The  first  AEG  and  the  first  ABOEG 
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classes  received  version  A  as  pre-test  and  version  B  as  post-test;  the  second  ALC  and  the  second 
ABOLC  classes  received  version  B  as  pre-test  and  version  A  as  post-test. 

ALC  students  comprised  the  majority  of  enlisted  participants;  83  of  the  84  enlisted 
participants  were  ALC  students  (99%).  Among  ALC  students,  order  was  roughly  balanced:  45 
were  administered  Exam  A  as  the  pre-test  (54%)  and  38  were  administered  Exam  B  as  the  pre¬ 
test  (46%).  Among  AEC  students,  time  in  service  ranged  from  0  to  20  years,  mean  =  7.5  (SD  = 
2.70).  Enlisted  deployment  frequency  ranged  from  0  to  6  deployments,  mean  =  2.5  (SD  =  1.05). 
Among  enlisted  personnel,  all  but  one  held  variants  of  the  1 1  series  Infantry  MOS,  with  the 
remaining  enlisted  NCO  having  been  cadre  within  an  ABOEC  class.  All  NCOs  were  E5  and  E6 
rank.  Enlisted  age  ranged  from  22  to  39,  mean  =  27.9  (SD  =  3.24).  All  enlisted  participants 
were  male. 

ABOEC  students  accounted  for  all  commissioned  officer  participants,  with  a  total  of  91 
included  in  the  study.  Among  ABOEC  students,  counterbalance  order  was  nearly  equal,  with  45 
administered  Exam  A  as  the  pre-test  (49%)  and  47  administered  Exam  B  as  the  pre-test  (51%). 
Officer  time  in  service  ranged  from  0  to  18  years,  mean  =  2.4,  (SD  =  3.15).  Officer  deployment 
frequency  ranged  from  0  to  4,  with  82  officers  (90%)  never  having  deployed.  Eighty-four  (92%) 
officers  held  a  variation  of  the  19  series  (Armor  or  Cavalry)  MOS,  with  one  officer  enrolled  in 
Officer  Candidate  School,  four  18  series  Special  Eorces  officers,  one  military  intelligence 
officer,  and  one  officer  of  unspecified  MOS.  All  officers  were  of  rank  1st  Eieutenant  or  2nd 
Eieutenant.  Officer  age  ranged  from  22  to  36,  mean  =  24.3  (SD  =  3.06).  All  participating 
officers  were  male. 

Test  item  characteristics.  Responses  to  the  video  clips  were  evaluated  for  internal 
consistency  looking  at  positive  scores  (hits),  negative  scores  (false  positives),  and  overall  scores 
(cumulative  scores)  for  each  item.  Eor  the  purposes  of  reliability  analyses,  data  were  included 
from  all  respondents  who  completed  each  exam.  A  total  of  248  participants  completed  Exam  A, 
and  246  participants  completed  Exam  B. 

Eor  both  exams,  reliability  was  calculated  when  including  the  hit  responses  to  each  of  the 
three  questions  associated  with  each  of  the  eight  vignettes,  independent  of  false  positive 
responses,  resulting  in  a  total  of  24  items.  Reliability  was  then  calculated  including  false 
positive  responses  independent  of  hit  responses.  Einally,  reliability  calculations  were  conducted 
including  all  item  responses.  High  levels  of  internal  consistency  were  not  necessarily  expected 
(cf.  Mallery,  2003),  as  the  video  vignette  examinations  were  not  designed  to  be  a  unidimensional 
measure  of  aggression  detection  abilities,  but  rather  a  cross-sectional  capture  of  student  ability  to 
detect  threats  through  the  detection  of  numerous  cues  in  a  variety  of  contexts. 

Exam  A.  Hits  exhibited  an  intermediate  level  of  internal  consistency  (Cronbach’s  a  = 
.43).  Exam  A  false  positives  exhibited  a  similar  level  of  internal  consistency  (Cronbach’s  a  = 
.40).  Internal  consistency  did  not  substantively  change  when  including  both  types  of  responses 
(Cronbach’s  a  =  .39). 

Exam  B.  Internal  consistency  levels  within  Exam  B  were  roughly  equivalent  to  those 
observed  among  Exam  A.  Hit  items  exhibited  an  “unacceptable”  level  of  internal  consistency 
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(Cronbach’s  a  =.38).  False  positive  items  exhibited  similarly  low  levels  of  internal  consistency 
(Cronbach’s  a  =  .40),  as  did  all  items  when  considered  simultaneously  (a  =.36). 

Overall  performance.  The  primary  analysis  of  performance  collapsed  across  all  response 
scores  to  all  items  for  all  vignettes,  resulting  in  a  single  score  for  each  participant  for  each  exam. 
False  positives  were  coded  with  their  assigned  negative  values,  and  hits  with  their  assigned 
positive  values.  An  Analysis  of  Variance  (ANOVA)  was  conducted  in  order  to  test  for  student 
improvement  in  video  exam  performance  as  a  result  of  exposure  to  the  ASAT  course.  Exam, 
including  scores  from  Exam  A  and  Exam  B,  was  included  as  a  repeated,  within- subjects  factor. 
Administration  order  (A  or  B  pre-instruction)  and  rank  (enlisted  vs.  officer)  were  both  included 
as  between  subjects  factors.  Exam  was  significantly  related  to  performance,  Exam  B  scores 
(mean  =  36.29,  SD  =  11.97)  were  significantly  higher  than  were  Exam  A  scores  (mean  =  22.12, 
SD=  11.46),  E(l, 171)  =  164.14,  r|p^  =  .49).  The  effect  of  exam  was  qualified  by  an  exam  by 
exam  order  interaction,  E(l,  171)  =  21.38,  p  <  .001,  =  .11.  Cicchetti  interaction  post-hoc 

tests  revealed  that,  whereas  Exam  A  scores  reflected  no  difference  between  pre-course 
administration  (mean  =  21.48,  SD  =  10.48)  and  post-course  administration  (mean  =  22.78,  SD  = 
11.46),  Exam  B  post-course  administration  scores  (mean  =  40.48,  SD  =  11.41)  were  significantly 
higher  than  pre-course  scores  (mean  =  31.85,  SD  =  10.96,  p  <  .05);  see  Eigure  4. 


■  Pre  Course  □  Post  Course 


40.48 


Figure  4.  Video  vignette  overall  performance.  Scores  reflect  aggregate  of  hits  and  false 
positives. 


Additional  Analyses.  A  set  of  four  ANOVAs  was  conducted  to  evaluate  changes  on 
exam  performance  from  pre-  to  post-course.  Aggregate  scores  for  hits,  false  positives,  heuristic 
items,  and  detail  items  were  calculated  and  each  used  as  dependent  variables  in  the  four  separate 


19 


analyses.  For  each  analysis,  the  independent  factors  mirror  those  included  in  the  overall  score 
analysis.  The  Holms-Bonferroni  correction  for  familywise  error  was  applied  for  the 
interpretation  of  all  main  effects,  interactions,  and  post  hoc  tests.  (Holms,  1979).  All  reported 
values  include  application  of  the  Holms-Bonferroni  correction.  Only  significant  main  effects 
and  interactions  are  reported;  significant  interactions  are  presented  in  detail. 

Hits.  The  analysis  of  positive  responses  revealed  main  effects  for  exam  and  exam  order 
driven  by  two  separate  interactions.  Significant  interactions  were  observed  between  exam  and 
exam  order,  F(l,  171)  =  61 31 ,  p  <  .01,  =  .28  and  between  exam  order  and  rank  (enlisted  vs. 

officer),  F(l,  171)  =  7.42,  p  <  .05,  pp^  =  .042.  Cicchetti  interaction  post  hoc  tests  were 
conducted  to  test  for  specific  differences  in  the  cases  of  both  interactions.  Post  hoc  tests  of  the 
exam  by  administration  order  interaction  revealed  that  whereas  there  was  no  difference  in  scores 
by  order  within  Exam  A,  Exam  B  responses  were  higher  when  administered  post-course  (mean  = 
55.03,  SD  =  7.97)  than  when  administered  pre-course  (mean  =  45.44,  SD  =  931,  p  <  .05;  Eigure 
2).  Post  hoc  tests  of  the  administration  order  by  rank  interaction  identified  a  test  base  that,  as  a 
whole,  performed  more  poorly  than  all  other  groups.  The  overall  positive  response  performance 
of  ABOEC  officers  who  were  administered  Exam  B  pre-course  and  Exam  A  post-course  (mean  = 
42.82,  SD  =  1.01)  was  significantly  lower  than  scores  of  NCOs  exposed  to  the  same  exam  order 
(mean  =  47.00,  SD  =  1.10)  and  officers  who  were  administered  Exam  A  pre-course  and  Exam  B 
post-course  (mean  =  48.52,  =1.02). 


■  Pre  Course  □  Post  Course 


44.04 


55.03 


Figure  5.  Hits  performance;  includes  both  officer  and  enlisted  results. 
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■  Enlisted  □  Officer 


48.52 


Figure  6.  Officer  vs.  enlisted  performance.  Order  1  is  A  then  B;  Order  2  is  B  then  A. 


False  Positives.  The  analysis  of  negative  responses  revealed  a  main  effect  for  exam 
qualified  by  an  exam  by  administration  order  interaction,  ^(1,  171)  =  6.58,  p  <  .05,  =  .037. 

Cicchetti  interaction  post  hoc  tests  were  conducted  to  test  for  specific  differences  between  cells 
of  the  exam  by  administration  order  interaction.  Post-tests  for  both  administration  orders 
indicated  that  the  extent  of  false  positive  responses  observed  for  Exam  A  was  significantly 
greater  than  the  extent  of  false  positive  responses  for  Exam  B  (Table  10).  Whereas  for  both 
Exams  A  and  B,  false  positive  means  were  higher  when  the  exam  was  administered  as  post¬ 
course  in  comparison  to  when  that  exam  was  administered  pre-course,  neither  of  these 
differences  was  significant. 


Table  10 

Negative  Responses-Exam  by  Administration  Order 


Order  A  then  B 

Order  B  then  A 

Exam  A 

-19.06  (6.15) 

-21.25  (6.32) 

Exam  B 

-14.55  (6.27) 

-13.58  (5.95) 

Note.  Cells  with  matching  superscripts  do  not  differ  significantly;  those  without  matching  superscripts 


differ  at  the  p  <  .05  level. 


Heuristic  performance.  The  analysis  of  performance  on  the  heuristic  exam  subscales 
revealed  main  effects  for  both  exam  and  administration  order,  qualified  by  an  exam  by  order 
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interaction,  F{1,  171)  =  14.23,/)  <  .01,  =  .08.  Cicchetti  interaction  post  hoc  tests  revealed 

that  Exam  B  scores  were  significantly  higher  when  the  exam  was  administered  post-course 
(mean  =  22.02,  SD  =  8.41)  compared  to  pre-course  administration  (mean  =  15.07,  SD  =  9.29,  p  < 
.05).  There  was  no  observed  difference  in  performance  on  Exam  A  on  the  heuristic  subscale  by 
administration  order  (Eigure  7).; 


■  Pre  Course  □  Post  Course 


22.02 


Exam  A  Exam  B 


Figure  7.  Performance  for  heuristic  items. 


Detail  performance.  The  analysis  of  performance  on  the  detail  exam  subscales  revealed 
a  main  effect  for  exam  qualified  by  an  exam  by  administration  order  interaction,  E(l,  171)  = 
9.41,  p  <  .05,  r|p^  =  .052.  Post-tests  revealed  that,  whereas  there  were  no  significant  differences 
by  administration  order  within  each  exam.  Exam  A  and  Exam  B  both  exhibited  higher  means 
when  administered  post-course  (Table  11  and  Eigure  8).  Exam  B  scores  were  significantly 
higher  than  were  Exam  A  scores  in  both  administration  orders. 

Table  11 

Detail  Subscales-Exam  by  Administration  Order 


Order  A  then  B 

Order  B  then  A 

Exam  A 

9.34  (4.34) 

10.73  (5.28) 

Exam  B 

18.46  (6.10) 

16.78  (5.48) 

Note.  Cells  with  matching  superscripts  do  not  differ  significantly,  those  without  matching  superscripts 
differ  at  the  p  <  .05  level. 
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■  Pre  Course 


□  Post  Course 


18.46 

16.78  I - 


Exam  A  Exam  B 


Figure  8.  Performance  on  detail  items.  Performance  was  significantly  greater 
on  post- test  for  both  Exam  A  and  Exam  B. 


Discussion 

The  collective  evidence  viewed  over  all  evaluations  indicates  that  the  ASAT  program 
improves  Soldier  ability  to  identify  pre-event  indicators  prior  to  violent  action  through  the  use  of 
combat  profiling  techniques.  Soldier  AAR  responses  were  overwhelmingly  positive  among  both 
combat  veterans  and  those  Soldiers  who  have  yet  to  deploy.  The  declarative  knowledge  tests 
demonstrated  that  student  knowledge  of  ASAT  curriculum  significantly  increases  after  exposure 
to  the  course.  The  video  SJT  examinations  suggested  that  students  are  better  able  to  identify 
details  and  scenarios  from  a  law  enforcement  environment  without  a  significant  increase  in  false 
assertions  after  exposure  to  the  course.  However,  the  interpretation  of  the  video  SJTs’  efficacy 
must  be  tempered  by  the  fact  that  Exam  A  was,  overall,  more  difficult  than  Exam  B.  This  lack  of 
parallelism  between  the  two  exams  may  have  been  due  to  their  content:  Exam  A  had  over¬ 
representation  of  “Police  Stops”  while  Exam  B  had  over-representation  of  “Store  Eootage.” 

Evidence  of  the  ability  of  ASAT  to  provide  far  transfer  in  detecting  indicators  of 
imminent  aggression  is  provided  by  the  results  of  the  video  examination.  The  video  examination 
used  short  segments  of  law  enforcement  and  surveillance  scenarios,  none  longer  than  2  minutes 
in  length,  to  test  student  ability  to  engage  in  combat  profiling  and  correctly  identify  scenarios  and 
details.  Clips  were  also  of  generally  poor  quality  with  regards  to  camera  perspective  and 
resolution.  The  nature  of  the  test  placed  students  at  an  immediate  disadvantage  in  that: 

•  the  tested  subject  domain  significantly  diverged  from  the  Iraq/ Afghanistan  focus  of  the 

ASAT  course; 

•  clip  length  prevented  students  from  developing  a  scenario  baseline  as  taught  in  ASAT; 

•  poor  clip  quality  precluded  highly  detailed  observation  of  some  content;  and 
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•  video  clips  eliminated  all  sounds,  smells,  and  other  “atmospherics”  taught  as  valuable 

observation  tools  in  ASAT. 

In  spite  of  the  handicaps  introduced  by  the  nature  of  the  video  exams,  students  still 
exhibited  increases  in  their  ability  to  accurately  interpret  the  post-test  video  clip  scenarios  and 
exhibited  only  small,  non- significant  increases  in  false  positives  on  this  test  when  comparing  pre 
and  post-course  administrations.  The  implication  is  that  students  did  not  engage  in  more 
guessing  behavior,  nor  did  they  adopt  a  false  confidence  in  their  observations  skills,  but  rather 
that  the  ASAT  course  gave  the  students  the  ability  to  more  accurately  observe  the  operational 
environment. 

One  of  the  more  intriguing  issues  in  the  assessment  of  the  ASAT  program  was  to 
determine  what  skills  are  taught  during  ASAT  versus  what  skills  are  learned.  An  extensive,  in- 
depth  review  of  the  nonverbal  behavior  literature  revealed  no  overlap  with  the  majority  of  the 
content  of  the  ASAT  course.  The  most  significant  overlap  witnessed  was  in  the  “gray  literature,” 
among  books  written  by  former  combat  veterans  and  US  Federal  law  enforcement  agents. 
Interpretation  of  the  content  was  further  complicated  by  much  of  the  ASAT  curriculum  that  is 
possibly  included  not  necessarily  for  its  value  in  conveying  fact  but  for  means  of  general 
emphasis  on  the  importance  of  observation.  For  example,  the  kinesics  module  of  the  course 
includes  a  component  focused  on  the  detection  of  deception.  During  this  segment,  ASAT 
instructors  indicate  that  individuals  about  to  provide  truthful  statements  will  “rub  their  foreheads, 
because  their  prefrontal  cortexes  heat  up  because  they  are  thinking.”  Whereas  this  point  is  based 
in  the  loosest  interpretation  of  psychophysiology/neuroanatomy  and  incorrect,  the  teaching  point 
is  likely  intended  simply  as  emphasis  on  observing  a  target  for  external  signs  of  cognition  and/or 
anxiety. 

It  is  clear  that  combat  profiling  is  far  more  art  than  science,  based  on  the  collective 
experience  of  combat  veterans,  law  enforcement  personnel,  and  big  game  hunters.  The 
underlying  principles  are  mentioned  in  Army  doctrine  but  generally  scattered  across  multiple 
manuals  and  guides.  The  practices  are  relatively  familiar  to  reconnaissance  units  and  snipers,  but 
generally  foreign  to  most  other  Army  personnel.  Given  the  current  understanding  of  the  ASAT 
program,  the  most  correct  description  of  the  program  is  that  it  facilitates  Soldiers’  ability  to 
interpret  nonverbal  indicators  of  imminent  aggression. 

The  results,  especially  those  of  the  video  examination,  support  that  ASAT  teaches  skills 
that  are  applicable  across  multiple  domains  of  Army  operations.  Operation  Enduring  Freedom 
was  almost  exclusively  a  counterinsurgency  operation,  and  the  skills  taught  in  ASAT  are  most 
valuable  in  counterinsurgency  and  policing  operations.  A  continued  emphasis  on  training 
Soldiers  to  accurately  interpret  nonverbal  behaviors  will  prove  valuable  as  US  forces  continue 
current  and  future  counterinsurgency  operations. 

Consistent  with  previous  limited  work  on  video  based  SJTs,  the  research  successfully 
produced  and  demonstrated  the  feasibility  of  video-based  examinations  of  socially  based  tasks. 
The  tests  and  procedures  used  in  this  investigation  are  in  need  of  more  refinement  and  would 
benefit  greatly  from  higher  quality  stimuli.  Despite  their  shortcomings,  they  still  detected 
improvement  in  a  domain  that  is  very  difficult  to  capture.  We  believe  that  the  exams  produced  in 
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this  investigation  can  serve  as  a  precursor  model  for  future  efforts  to  test  qualitative  skills  in  a 
variety  of  settings. 
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Appendix 


Overview  of  ASAT  Course  Material 


ASAT  training  does  not  teach  a  new  approach  to  situation  awareness  (SA)  information 
management  or  integration,  nor  does  it  espouse  a  new  model  of  understanding  SA.  What  ASAT 
training  does  attempt  is  to  provide  Soldiers  with  a  set  of  goals,  objectives,  observation  tools,  and 
expectations  with  which  to  perceive  the  operational  environment  (Level  1)  along  with  the 
knowledge  base  and  cognitive  skills  necessary  to  comprehend  incoming  information  (Level  2) 
and  make  projections  (Level  3)  that  influence  the  military  decision  making  process  and 
subsequent  actions.  At  its  core,  ASAT  training  is  intended  to  provide  Soldiers  with  the  ability  to 
interpret  the  “language”  of  nonverbal  behavior  in  the  context  of  the  operational  environment. 
The  ASAT  course  includes  instructional  modules  that  focus  on  domains  of  combat  profiling, 
combat  multipliers,  the  terrorist  planning  cycle,  and  advanced  observation  techniques.  The 
majority  of  the  ASAT  course  focuses  on  combat  profiling  techniques. 


Ii^antry-FocusedS 


Enemy  Location/Actions/Status  Shetter/Protection  Obstacles 
Friendly  Location/Actions/Status  Visibility  ■  Terrain 

Civilian  Location/Acbons/Status  H  Weather 


SA  Devices 


Figure  Al.  Model  of  Infantry  SA  (from  Endsley  et  ah,  2000 


Combat  Profiling 


Combat  profiling  refers  to  the  ASAT  system  of  observing  and  interpreting  the  nonverbal 
behaviors  of  combatants  and  civilians  in  an  operational  environment.  The  ASAT  combat 
profiling  module  is  divided  into  six  distinct  “domains,”  each  of  which  addresses  a  separate 
aspect  of  nonverbal  behavior.  These  domains  include  biometrics,  kinesics,  proxemics. 
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geographies,  heuristics,  and  atmospherics.  The  unifying  theme  of  all  six  domains  is  the 
identification  of  “anomalies,”  described  as  events  standing  out  from  the  environmental  baseline. 
Students  are  taught  to  focus  on  nonverbal  behavior  as  a  means  of  identifying  shifts  from  baseline 
that  can  serve  as  pre-event  indicators.  All  behaviors  that  are  the  focus  of  the  AS  AT  curriculum 
are  described  as  universally  displayed  and  interpreted  across  cultures  and  difficult  or  impossible 
to  intentionally  control  or  manipulate. 

The  six  domains  are  outlined  below. 

a.  Biometrics.  Biometrics  refers  to  observable  physiological  reactions  that  correspond 
to  internal/mental  states.  The  training  focuses  on  pupil  dilation,  “histamine  flush” 
(blushing),  and  sweating  as  a  means  of  detecting  arousal.  Particular  attention  is  paid 
to  detecting  these  indicators  among  isolated  individuals  within  an  environment  as  a 
means  of  identifying  individuals  who  are  in  a  heightened  state  of  arousal  while  those 
around  them  are  not.  Identifying  individuals  among  which  high  levels  of  arousal  do 
not  correspond  to  that  of  those  in  the  immediate  vicinity  can  facilitate  detection  of 
pre-event  indicators.  The  use  of  thermal  sights  to  detect  the  thermal  signatures  of 
targets  is  included  in  the  biometrics  domain. 


b.  Kinesics.  Kinesics  refers  to  culturally  universal  nonverbal  behavior  that  corresponds 
to  a  target’s  internal  state.  The  kinesics  domain  focuses  on  four  observable 
behavioral  clusters:  group  inclusion  cues,  deception  indicators,  aggression  indicators, 
and  displays  of  dominance. 

Group  inclusion  cues  are  a  cluster  of  nonverbal  behaviors  that  indicate  both  the 
focus  of  a  target’s  attention  during  a  social  interaction  and  if  that  interaction  is  open 
to  others.  Stance  with  regards  to  foot  direction  is  taught  as  a  significant  indicator  of 
the  exclusivity  of  a  conversation;  in  exclusive  two-person  conversations,  each 
individual  will  point  directly  towards  the  other  conversant  forming  a  “box”  which 
discourages  other  individuals  from  attempting  to  enter  the  conversation.  Another  sign 
of  exclusion  is  “blading,”  where  conversants  turn  their  backs  (shoulder  blades)  to 
those  that  might  attempt  to  join  the  conversation.  Open  conversations  can  be 
observed  as  each  conversant  has  one  foot  pointed  towards  the  other  conversant  and 
the  other  pointed  out,  opening  a  “triangle”  as  an  invitation  for  others  to  join  the 
conversation.  Generally,  throughout  the  conversation,  posture  and  direction  of  foot 
pointing  will  indicate  the  target  of  attention.  Other  signs  of  inclusion  include  head 
nodding  and  hand  gestures  that  sweep  in  towards  the  chest  of  the  speaker. 

Deception  indicators  are  taught  as  a  cluster  of  behaviors  may  be  exhibited  as  a 
target  speaks.  These  behaviors  include  a  target  looking  down  and  to  the  left  when 
talking,  touching  his  or  her  nose,  a  significant  increase  in  blink  rate,  automatic 
nodding  of  the  head  that  indicates  agreement  or  disagreement  with  a  statement, 
licking  of  the  lips,  and  rubbing  the  ears.  Also,  a  “histamine  flush”  as  discussed  in  the 
biometrics  domain  may  accompany  these  indicators,  as  physiological  arousal  may 
accompany  deception.  Rubbing  of  the  forehead  is  described  as  a  pre-statement 
indicator  that  a  target  is  about  to  tell  the  truth. 
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Aggression  indicators  are  taught  as  a  cluster  of  behaviors  associated  with  an 
aggressive  internal  state  and  are  often  indicators  of  an  impending  attack.  Facial 
indicators  noted  include  flared  nostrils,  clenching  of  the  eyebrows  and  closing  of  the 
eyes  in  disagreement  during  an  interaction.  Physical  indicators  include  crossed  arms 
and  balled  fists,  along  with  histamine  flush  as  an  indicator  of  arousal. 

Displays  of  dominance  comprise  several  types  of  nonverbal  behaviors.  The  most 
often  discussed  and  most  common  is  the  target  stance  of  hands  on  hips.  This  makes 
the  individual  appear  larger,  and  emphasizes  a  target’s  sex  through  the  placement  of 
hands.  Males  and  females  are  described  as  modifying  this  stance  to  point  toward 
their  respective  reproductive  organs:  males  towards  the  front  of  the  pelvis,  females 
towards  the  rear.  During  handshakes,  the  rotation  of  a  target’s  hand  a  quarter  turn 
counterclockwise,  over  the  hand  of  the  other  individual  is  described  as  a  sign  of 
dominance.  Crossed  arms  and  balled  fists  are  both  sited  as  signs  of  aggression  or 
discontent.  A  target  turning  his  or  her  head  to  the  side  to  reveal  the  jugular  area  is 
described  as  a  sign  of  submission.  Additionally,  a  readily  recognizable  “predatory 
look”  is  described  that  is  often  accompanied  with  a  “mission  focus”  that  can  be  used 
as  a  means  of  recognizing  the  precursors  of  an  imminent  attack. 

c.  Proxemics.  Proxemics  is  a  focus  on  relative  spatial  distances  between  individuals  as 
a  means  of  evaluating  targets  within  the  environment.  Training  in  the  proxemics 
domain  comprises  three  focus  areas:  High  Value  Individual/Target  (HVFHVT) 
indicators,  the  significance  of  proxemic  pushes  and  pulls,  and  the  tactical  importance 
of  standoff  (distance  from  a  target). 

HVT  identification  is  often  the  goal  of  surveillance  efforts  and  necessary  prior  to  a 
high  proportion  of  aerial  strikes  and  ground  raids.  Identification  of  an  HVT  is  not 
always  possible  through  facial  recognition  or  the  presence  of  a  uniform,  particularly 
when  little  intelligence  is  available  regarding  a  target.  ASAT  teaches  a  cluster  of 
indicant  behaviors  that,  when  conducted  in  close  physical  proximity  to  a  target, 
indicate  HVT  or  HVI  status  for  that  target.  The  acronym  MADE,  representing 
mimicry,  adoration  (positive  or  negative),  direction,  and  entourage  (the  presence  of) 
is  taught  to  students  as  a  mnemonic  for  identifying  a  target  as  a  HVT.  The 
observation  of  multiple  indicators  directed  towards  a  single  target  signifies  the 
possibility  that  a  target  is  an  HVT. 

Proxemic  pushes  and  pulls  are  described  as  natural  approaches  and  repulsions 
towards  individuals  and  objects  in  the  environment  that  are  visible  during  routine 
reconnaissance  and  patrolling.  Simply  described,  individuals  are  naturally  attracted 
to  things  that  they  like  or  feel  comfortable  around  and  repulsed  from  those  things  that 
they  dislike  or  become  uncomfortable  in  the  presence  of.  Observing  pushes  is  a 
particularly  effective  means  of  detecting  shifts  from  baseline,  as  a  local  crowd  will 
often  exhibit  a  proxemic  push  and  keep  an  unusual  distance  from  an  insurgent  about 
to  carry  out  an  attack  on  coalition  forces. 
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d.  Geographies.  As  taught  in  ASAT,  geographies  describes  a  means  of  perceiving  the 
terrain  in  terms  of  the  people  and  social  groups  that  frequent  an  area.  Three  general 
concepts  are  communicated  throughout  this  module-natural  lines  of  drift,  anchor 
areas,  and  habitual  areas. 

Natural  lines  of  drift  are  described  in  multiple  land  navigation  training  courses 
and  manuals  as  areas  that  are,  by  terrain  characteristics,  more  easily  traversed  than  are 
surrounding  areas.  In  these  areas,  less  thick  underbrush,  downhill  slopes,  or  more 
even  ground  lead  humans  and  animals  traveling  in  the  area  to  gravitate  towards  and 
stay  in  these  natural  lanes.  In  the  ASAT  curriculum,  the  natural  line  of  drift  is  framed 
in  terms  of  an  urban  or  town  setting  and  in  identifying  where  people  tend  to  move 
while  they  travel  within  an  urban  setting  in  order  to  better  identify  when  an  individual 
may  be  out  of  a  baseline  routine. 

Anchor  areas  are  frequented  by  members  of  a  specific  group.  These  areas  are 
described  as  controlled  and  inhabited  only  by  members  and  affiliates  of  a  specific 
group  and  often  marked  by  significant  iconography  (see  atmospherics,  below).  Once 
identified,  anchor  points  not  only  serve  to  designate  individuals  at  a  particular 
location  as  group  members,  but  can  help  predict  post- attack  travel  routes  when 
considered  along  with  natural  lines  of  drift. 

Habitual  areas  are  described  as  areas  frequented  by  many  individuals  from 
multiple  groups.  These  areas  include  markets,  roads,  town  centers,  and  other  areas 
generally  considered  public  and  that  frequently  solicit  crowds. 


e.  Heuristics.  Heuristics  refers  to  a  process  of  interpreting  observations  by  matching 
them  to  mental  models  developed  through  experience,  education,  or  shared 
knowledge.  SMEs  describe  heuristic  matches  as,  “tactical  shortcuts,”  through  which 
enemy  tasks  such  as  reconnaissance,  bounding  overwatch,  infiltration,  and  sniper 
emplacement  can  be  quickly  and  efficiently  identified.  The  use  of  the  heuristics 
domain  is  described  as  a  game  of  charades,  in  which  observations  of  excerpts  from  a 
series  of  events  can  be  used  to  extrapolate  the  entirety  of  a  target’s  actions  and 
intentions. 

Heuristics  are  described  as  used  both  as  a  means  of  detecting  attacks  and  as  a 
means  of  perpetrating  attacks.  Social  camouflage  is  a  method  by  which  insurgents 
can  use  the  local  populace  as  a  means  of  obscuring  offensive  operations.  This 
technique  is  utilized  when  common  dress  and  tasks,  such  as  building  maintenance 
carried  out  by  laborers,  are  used  to  obscure  items  and  operations.  For  example,  an 
insurgent  two-man  sniper  team  using  social  camouflage  would  take  on  the  dress  and 
tools  of  men  gathering  firewood  in  order  to  scout  possible  positions  and  emplace 
while  carrying  their  equipment  hidden  within  these  tools.  A  related  technique,  urban 
masking,  is  commonly  used  to  conceal  lEDs,  and  involves  placing  weapons  in  the 
environment  that  are  cloaked  as  mundane  items,  such  as  roadside  trash  or  a  vehicle 
that  has  run  out  of  fuel. 
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f.  Atmospherics.  Atmospherics  refers  to  the  gestalt  of  the  information  present  in  an  area 
at  any  given  period  of  time.  This  information  includes  all  five  senses,  with  a  focus  on 
sights,  sounds,  and  smells.  Atmospherics  is  taught  as  a  means  of  detecting  pre-event 
indicators  by  establishing  a  baseline  for  events  that  occur  within  an  area,  such  as  call 
to  prayer  and  the  opening  of  storefronts.  The  process  of  “baselining”  an  area  can 
allow  trained  Soldiers  to  detect  when  something  unusual  has  occurred  or  is  about  to 
occur. 

Iconography  is  included  in  the  instruction  for  the  atmospherics  domain  of  combat 
profiling.  Iconography  includes  all  aspects  of  dress,  adornment,  and  building  or 
structure  labeling  (specifically  including  graffiti)  that  indicates  group  allegiance  and 
ideology. 

Five  Combat  Multipliers 

Combat  multipliers  are  defined  as  actions  that  can  increase  the  likelihood  of  mission 
success  in  the  absence  of  changes  to  the  size  of  existing  units  (FM  101-5-1,  1997).  ASAT 
curriculum  includes  instruction  on  the  use  of  five  “combat  multipliers”  that  do  not  explicitly 
appear  in  Army  doctrine.  The  ASAT  “combat  multipliers”  are  described  as  practices  that 
maximize  the  impact  of  the  skills  taught  during  the  course.  In  the  following  sections,  each 
combat  multiplier  is  described  and  discussed  in  terms  of  supporting  doctrine. 

1.  Tactical  Cunning.  ASAT  describes  tactical  cunning  as  the  ability  to  “see  the  battle 
space  through  the  enemy’s  eyes”  by  using  combat  profiling  techniques  to  recognize  and  correctly 
interpret  pre-event  indicators.  Perspective  taking  is  integral  to  this  ability;  students  are  taught  to 
project  enemy  objectives  and  intentions  in  order  to  determine  both  intended  final  outcomes  for 
an  enemy  operation  and  an  operation’s  current  stage  of  execution  in  the  terrorist  planning 
process.  Tactical  cunning  is  defined  in  FM  3-24  (U.S.  Army,  2006)  Chapter  7,  paragraph  6  as, 
“the  art  of  employing  fundamental  skills  of  the  profession  in  shrewd  and  crafty  ways  to  out-think 
and  out- adapt  enemies.”  Emphasis  is  placed  on  the  training  of  all  levels  of  leadership  in  this  skill 
in  FM  3-24. 

2.  Tactical  Patience.  ASAT  describes  tactical  patience  as  “speeding  up  or  slowing  down 
the  battlefield”  to  a  unit’s  advantage.  The  ASAT  program  teaches  that  increased  situational 
awareness  of  an  AO  (tactical  cunning)  resulting  from  combat  profiling  allows  combat  leaders  to 
anticipate  significant  events.  Tactical  patience  is  exercised  when  a  leader  waits  for  conditions  to 
be  favorable  for  mission  success  prior  to  initiating  actions.  As  taught  by  ASAT,  the  improve¬ 
ments  to  small  unit  situational  awareness  resulting  from  combat  profiling  can  be  used  to  improve 
tactical  patience;  knowing  enemy  intent  can  lead  to  execution  of  well-timed  operations  that  deny 
the  enemy  offensive  operations  and  effective  defensive  positions.  The  term  tactical  patience  is 
not  explicitly  defined  within  Army  doctrine,  but  appears  in  Army  FM  7-15  (U.S.  Army,  2009) 
and  is  commonly  used  in  numerous  training  environments. 

3.  Geometry  offices.  The  ASAT  program  teaches  “geometry  of  fires’  as  a  metaphor  to 
illustrate  the  concept  of  multi-faceted  intelligence  collection.  Geometry  of  fires  refers  to 
overlapping  sectors  of  fire-an  integral  concept  in  small  unit  tactics,  particularly  when  occupying 
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defensive  positions.  ASAT  applies  the  concept  of  360°  sector  security  and  overlapping  sectors 
of  fire  to  intelligence  gathering,  stressing  the  importance  of  careful  coordination  of  intelligence 
gathering  efforts.  ASAT  teaches  that  every  Soldier  can  gather  information  through  combat 
profiling  that  may  be  integral  to  determining  enemy  actions,  but  that  carefully  planned 
intelligence  sharing  practices  are  necessary  in  order  to  consolidate  this  information  and  develop 
comprehensive  situational  awareness.  ASAT  training  indicates  that  every  Soldier  should  be 
taught  combat  profiling  techniques  and  tasked  with  reporting  perceived  critical  information  when 
it  is  detected.  Special  emphasis  is  placed  on  the  two-way  vertical  and  horizontal  flow  of 
information,  outlining  the  need  for  Soldiers  to  be  able  to  share  information  with  peers, 
subordinates,  and  superiors  in  real  time. 

ASAT  training  includes  special  emphasis  on  the  use  of  Observation  Posts  (OPs)  on  the 
battlefield,  traditionally  conceived  of  as  a  tactical  position  intended  for  early  warning  and 
manned  by  a  fire-team  sized  element  (FM  3-21.8,  2007).  ASAT  teaches  that  OPs  should  exploit 
the  capabilities  of  specialized  optics  and  can  be  placed  anywhere  on  the  battlefield. 

4.  Guardian  Angel.  The  ASAT  concept  of  guardian  angel  directly  corresponds  to  the 
bounding  overwatch  movement  technique.  ASAT  emphasizes  the  use  of  overwatch  (guardian 
angel)  in  all  asymmetric  warfare  environments  to  enhance  security  and  situational  awareness. 
Overwatch  positions  are  able  to  observe  pre-event  indicators  that  maneuver  elements  cannot,  due 
to  their  perspective  of  the  battlefield,  offering  the  potential  to  detect  enemy  threats  prior  to 
execution.  Overwatch  positions  also  observe  events  that  occur  as  a  direct  reaction  to  the 
presence  of  a  maneuver  unit  and  can  often  detect  highly  valuable  diagnostic  indicators  that 
would  otherwise  go  unnoticed. 

5.  Good  Shepherd.  The  ASAT  concept  of  good  shepherd  directly  corresponds  to  the 
concept  of  winning  the  “hearts  and  minds”  of  the  local  populace.  This  is  integral  to 
counterinsurgency  doctrine  and  is  specifically  addressed  in  FM  3-24  (U.S.  Army,  2006). 
“Winning  hearts”  is  described  as  convincing  the  local  population  that  COIN  success  is  in  their 
best  interests.  “Winning  minds”  is  described  as  convincing  the  local  populace  that  COIN  forces 
are  both  able  to  offer  regional  security  and  unable  to  be  defeated.  Sensitivity  to  the  local 
populace  and  relationships  with  local  leaders  are  essential  to  achieving  both  of  these  goals. 

ASAT  also  describes  good  shepherd  as  a  means  of  positive  mentoring  within  the  small  unit  that 
facilitates  the  exchange  of  information  regarding  combat  profiling  techniques. 

The  Seven  Step  Terrorist  Planning  Cycle 

The  ASAT  program  teaches  a  seven  step  terrorist  planning  cycle  to  help  Soldiers 
interpret  information  about  the  enemy  perceived  through  combat  profiling.  After  enemy  forces 
are  identified  and  their  intentions  determined,  identifying  the  current  stage  of  their  efforts  can 
help  leaders  to  use  tactical  patience  in  order  to  maximize  the  effectiveness  of  counter  operations. 
Additionally,  knowledge  of  the  seven  step  model  may  help  Soldiers  make  an  “heuristic  match” 
when  observing  the  enemy  executing  a  step  in  the  cycle.  The  ASAT  model  of  the  terrorist 
planning  cycle  directly  corresponds  to  that  included  in  Army  doctrine  (TRADOC  G2  Handbook 
No.  1,  2007;  A  Military  Guide  to  Terrorism  in  the  Twenty-First  Century).  The  following  section 
provides  a  brief  description  of  each  step  as  described  by  the  ASAT  program.  These  descriptions 
are  altogether  consistent  with  Army  doctrine  and  no  content  comparison  is  included. 
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1.  Broad  Target  Selection.  When  planning  an  attack,  terrorists  will  develop  a  list  of 
potential  targets.  Potential  targets  are  selected  considering  location,  opportunities  for  attack, 
group  associations,  and  symbolic  value.  Soft  targets  are  the  most  desirable  and  most  commonly 
chosen,  as  they  maximize  likelihood  of  mission  success.  Consideration  is  given  to  target 
criticality  within  the  immediate  social  infrastructure,  casualties  that  can  be  inflicted  at  the  target 
site,  and  the  likely  public  attention  resulting  from  an  attack  on  each  target.  At  this  stage  of  the 
planning  process,  media,  internet,  and  other  sources  of  background  information  are  used  to 
screen  potential  targets.  Targets  meeting  the  needs  of  the  terrorist  mission  and  with  sufficient 
vulnerabilities  will  be  more  intensely  investigated  during  the  intelligence  and  surveillance  step. 

2.  Intelligence  and  Surveillance.  Once  a  list  of  targets  is  developed,  terrorists  will 
conduct  surveillance  and  gather  intelligence  on  each  potential  target.  The  information  collected 
for  each  potential  target  includes  a  focus  on  routines,  physical  layout,  security  measures,  and 
SOPs.  This  information  may  be  gathered  for  a  period  of  years  and  includes  primarily  passive 
means  of  intelligence  gathering. 

3.  Specific  Target  Selection.  Once  intelligence  collection  efforts  have  yielded  the 
necessary  information  regarding  all  potential  targets,  a  specific  target  is  selected  for  operational 
planning.  This  target  is  selected  considering  the  effects  of  a  successful  attack  with  regards  to  a 
larger  audience,  media  attention,  consistency  with  the  intended  political  statement,  larger  group 
objectives,  demonstration  of  the  group’s  capabilities,  and  a  cost/benefit  consideration.  Once  the 
most  desirable  target  is  selected,  pre-attack  surveillance  and  planning  will  begin,  although 
continued  surveillance  of  non-selected  targets  may  continue  if  they  are  deemed  to  be  potential 
targets  for  future  operations. 

4.  Pre-Attack  Surveillance  and  Planning.  Pre-attack  surveillance  is  often  carried  out  by 
trained  operatives  for  days  or  weeks  prior  to  an  attack.  During  this  phase,  target  weaknesses  and 
security  procedures  are  better  specified,  type  of  attack  is  decided,  recruitment  of  required 
specialized  operatives  takes  place,  and  preparatory  operations-including  establishing  a  base  of 
operations  and  escape  routes-are  conducted.  This  process  results  in  a  highly  specific  plan  of 
attack,  including  the  precise  means  and  intended  dates  of  plan  execution.  This  specific  attack 
plan  is  then  rehearsed  prior  to  execution. 

5.  Attack  Rehearsal.  The  attack  rehearsals  conducted  by  terrorist  groups  are  similar  to 
those  conducted  by  conventional  military  organizations.  Terrorists  rehearse  transportation, 
employment  of  weapons  systems,  and  coordinated  execution  of  the  mission.  Target  security 
reactions  are  often  closely  scrutinized  during  this  phase  as  a  form  of  advanced  surveillance,  with 
careful  attention  paid  to  security  force  reactions  to  specific  scenarios.  Often  operatives  are 
employed  to  probe  security  forces,  rehearse  infiltration  and  escape  routes,  and  to  confirm  target 
information  including  physical  layout,  patterns  of  activities,  and  security  force  capabilities. 
Rehearsals  provide  the  final  preparation  and  plan  revisions  directly  prior  to  execution.  Careful 
observation  on  the  part  of  security  forces  during  this  late  phase  of  preparation  can  often  detect  an 
imminent  terrorist  attack  and  may  even  reveal  the  intended  method  of  attack. 
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6.  Actions  on  Objective.  In  a  carefully  planned  terrorist  attack,  target  selection,  planning, 
and  rehearsal  will  optimally  exploit  target  weaknesses.  The  terrorist  will  possess  all  the 
advantages  of  initiative  by  employing  the  use  of  surprise,  setting  the  conditions  of  attack 
(including  time  and  location),  retaining  the  ability  to  use  both  diversions  and  follow-up  attacks, 
and  by  ensuring  the  opportunity  to  employ  both  security  and  support  positions.  After  completing 
the  attack,  terrorist  operatives  will  immediately  execute  the  escape  and  evasion  plan. 

7.  Escape  and  Evasion.  Escape  and  evasion  plans  are  universal  to  all  attacks  and 
generally  well  rehearsed.  In  the  case  of  suicide  attacks,  the  act  of  dying  during  the  attack  serves 
as  its  own  escape  plan  for  the  attacker,  but  the  personnel  supporting  the  attacker  will  require  their 
own  escape  and  evasion  plan.  Knowing  local  habitual  areas  and  group-specific  anchor  points 
can  lead  to  the  interception  of  terrorist  operatives  post-attack. 
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